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@ Image display apparatus and Image photographing apparatus therefor. 



@ This invention relates to an innage display ap- 
paratus having first image generating means for dis- 
N playing first image information by a plurality of pixels 
^ emitting light, second image generating means for 
^ displaying second Image information by a plurality of 
^ pixels emitting light, the second image information 
^ being part of the first image information, visual axis 
detecting means for detecting an observer's visual 
^ axis direction, optical means for combining a first 
^ image from the first image generating means and a 
O second image from the second image generating 
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means, moving the second image relative to the first 
image in conformity with an output signal from the 
visual axis detecting means and turning it to the 
observer's pupil, the marginal portion of the second 
Image being made substantially coincident with the 
boundary between the pixels of the first image when 
the images are combined, and Image display chang- 
ing means for changing the first image information 
and the second image information on the basis of 
the output signal of the visual axis detecting means. 
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BACKGROUND OF THE INVENTION 

Fjeld of the Invention 

This invention relates to an apparatus for dis- 
playing an image of a wide field angle, and particu- 
larly to an image display apparatus adapted to be 
mounted on the head portion or face of a person 
who observes an image and enable the person to 
observe the image information of a small display 
element such as a liquid crystal display element or 
a CRT therethrough while enlarging the image in- 
formation as a virtual image. 

Related Background Art 

A display apparatus using a multilayer film 
reflecting surface or an optically transparent beam 
coupling element such as a hologram optical ele- 
ment to enable display information (image informa- 
tion) from a display element and image information 
such as a scene in the external world to be spa- 
tially superposed one upon the other in the same 
field of view and observed is generally called a 
head up display apparatus and is utilized in various 
fields. Also, a display apparatus in which the beam 
combiner Is provided at a location relatively near to 
an observer's eyes In order to make the parts of 
the optical device of the head up display apparatus 
small and the optical device is mounted on a 
helmet is called a helmet mounted display appara- 
tus (HMD apparatus), and its various applications to 
a display apparatus for the operation of an aircraft, 
a display apparatus for amusements such as 
games or for virtual reality, etc. have been pro- 
posed. 

Fig. 1 of the accompanying drawings is a sche- 
matic view of an HMD apparatus as a display 
apparatus described in U.S. patent No. 4,028,725. 
Display lights emitted from CRTs 101 and 102 are 
made into substantially parallel beams of light by 
collimator lenses 103 and 108 and are reflected by 
mirrors 104 and 107, respectively, and the display 
lights from the two CRTs 101 and 102 are super- 
posed one upon the other by dichroic mirrors 105 
and 109. The display lights thus superposed one 
upon the other have their direction again changed 
by a mirror 106 and are enlarged through a lens 
110, and are directed to an observer's pupil 114 by 
a half mirror 111. At this time, the CRT 101 dis- 
plays a wide field image 113 and the CRT 102 
displays a narrow field image 112. Also, the dich- 
roic mirror 109 is installed so as to shield the 
vicinity of the center of the dichroic mirror 105 from 
light and therefore, the observer can observe the 
wide field image 113 displayed by the CRT 101 
and the narrow field image 112 from the CRT 102 
displayed near the center thereof while superpos- 



ing these images upon a scene or the like behind 
the half mirror 111. 

Also, in this example of the prior art, a so- 
called purkinje image provided by the light from a 

5 light source 115 of infrared rays being reflected by 
the observer's pupil 114 is detected by a detection 
unit 119 comprising a beam splitter 116 and 
photodiodes 117, 118, whereby the observer's vi- 
sual axis direction is detected. 

10 The output of this visual axis detecting means 

is inputted to the servo system (not shown) of the 
movable half mirror 1 1 1 and thus, the aforemen- 
tioned two field images are always displayed for- 
wardly in the observer's visual axis direction. 

75 In the example of the conventional art shown in 

Fig. 1, the field angles of the wide field image and 
the narrow field image are 25° and 5% respec- 
tively, and the narrow field image Is displayed with 
a high resolution, whereby with the displayed im- 

20 age of the high resolution as the center of the field 
of view, the wide field image is always observed 
around it. In the above-described example of the 
conventional art, however, design is made such 
that the image formed by the wide field image and 

25 the narrow field image being superposed one upon 
the other is always presented at the central portion 
of the field of view by the half mirror 1 1 1 controlled 
by the signal from the visual axis detecting means, 
and this has led to a problem that even if the 

30 observer gazes at any point on the wide field 
image, the image is not visually recognized at the 
center of the field of view and the portion intended 
by the observer cannot be observed. Also, a field 
angle much wider than the field angle as in the 

35 above-described example of the prior art, e.g. 25° , 
is required of the display apparatus for virtual re- 
ality and for example, a display for displaying a 
field angle of 88° by one eye has been commer- 
cialized, but in these apparatuses, the resolution of 

40 the displayed image depends on the pixel density 
of the liquid crystal display element used, and this 
has led to a disadvantage that at present, the 
displayed image does not have a sufficient resolu- 
tion. 

45 

SUMMARY OF THE INVENTION 

It is the object of the present invention to 
provide an image display apparatus for enabling an 

50 observer to observe the image of a place the 
observer wants to seen in a wide field image by a 
highly fine image, and an image photographing 
apparatus therefor. 

To achieve the above object, one form of the 

66 image display apparatus of the present invention is 
characterized by first image generating means for 
displaying first image information by a plurality of 
pixels emitting light, second image generating 
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means for displaying second innage information by 
a plurality of pixels emitting light, the second image 
information being part of the first image informa- 
tion, visual axis detecting means for detecting an 
observer's visual axis direction optical means for 
combining a first image from the first image gen- 
erating means and a second image from the sec- 
ond image generating means, moving the second 
image relative to the first image in conformity with 
an output signal from the visual axis detecting 
means and turning it to the observer's pupil, the 
marginal portion of the second image being made 
substantially coincident with the boundary between 
the pixels of said first image when the images are 
combined and image display changing means for 
changing said first image information and the sec- 
ond image information on the basis of the output 
signal of the visual axis detecting means. 

In a further preferred form, the size of the 
second image is integer times as great as the 
interval between the pixels (or the size of the pixel) 
of the first image when the images are combined. 

In still a further preferred form, the first image 
information in an area wherein the first image and 
the second image are superposed one upon the 
other when the images are combined is the second 
image information. 

In yet still a further preferred form, the area of 
the images superposed one upon the other is not 
displayed by the first image generating means. 

In a further preferred form, the pixel density in 
the central portion of the second image is higher 
than the pixel density of the first image and the 
pixel density in the marginal portion of the second 
image is substantially the same as the pixel density 
of the first image when said images are combined. 

In still a further preferred form, the luminance 
in the central portion of the second image is higher 
than the luminance of the first image and the 
luminance in the marginal portion of the second 
image is substantially the same as the luminance 
of the first image when the images are combined. 

In yet still a further form, the first and second 
images are combined on an intermediate imaging 
plane through respective relay lenses, are enlarged 
as virtual images and are turned to the observer's 
pupil. 

In a further preferred form, the relay lenses 
have different imaging magnifications. 

Another form of the image display apparatus of 
the present invention is characterized in that the 
image display apparatus is provided correspond- 
ingly to each of the observer's left and right eyes. 

In a further preferred form, the second image 
generating means corresponding to the left and 
right eyes display parallax image information cor- 
responding to the respective eyes. 



In still a further preferred form, the first image 
generating means is used in common. 

Another form of the image display apparatus of 
the present invention is characterized by first im- 

5 age generating means for emitting light and dis- 
playing first image information second image gen- 
erating means for emitting light and displaying sec- 
ond image information, the second image informa- 
tion being part of said first Image information visual 

10 axis detecting means for detecting an observer's 
visual axis direction head portion position detecting 
means for detecting the position of the observer's 
head portion optical means for combining a first 
image from the first image generating means and a 

75 second image from the second image generating 
means, moving the second image relative to the 
first image in conformity with an output signal from 
the visual axis detecting means and turning it to 
the observer's pupil and image information chang- 

20 ing means for changing the first image information 
and the second image information on the basis of 
the output signal of the visual axis detecting means 
and the output signal of the head portion position 
detecting means. 

25 A further preferred form has an image memory 

having recorded therein the image all around a 
predetermined position, and inputs a part of said 
image memory to said first image generating 
means as the first image information in conformity 

30 with the output signal of the head portion position 
detecting means. 

One form of the image photographing appara- 
tus of the present invention is an image photog- 
raphing apparatus for an image display apparatus 

35 for displaying, on the basis of output signals from 
visual axis detecting means for detecting the visual 
axis direction of an observer who observes a first 
image from first image generating means for dis- 
playing first image information and head portion 

40 position detecting means for detecting the position 
of the observer's head portion, a combined image 
of said first image and a second image from sec- 
ond image generating means for displaying second 
image information which is part of the first image 

45 information, characterized by first image pickup 
means for obtaining the first image information 
second image pickup means for obtaining the sec- 
ond image information means for controlling the 
photo-taking direction of the first image pickup 

50 means in conformity with the output signal from the 
head portion position detecting means and means 
for controlling the photo-taking direction of the sec- 
ond image pickup means in conformity with the 
output signal from the visual axis detecting means. 

55 A further preferred form has means for trans- 

mitting the image information obtained from the 
first image pickup means to the image display 
apparatus while making the compression rate of 
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the image information higher than that of the image 
information obtained from said second image pic- 
kup means. 

In still a further preferred form, the image 
photographing apparatus is provided correspond- 
ingly to each of the observer's left and right eyes. 

In yet still a further preferred form, said first 
image pickup means is used in common. 

Some specific embodiments of the image dis- 
play apparatus of the present invention will become 
apparent from the following detailed description 
thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 shows an example of the image display 
apparatus according to the conventional art. 

Fig. 2 shows a first embodiment of the Image 
display apparatus of the present invention. 

Figs. 3A, 3B, 3C and 3D are views for illustrat- 
ing images visually recognized by an observer. 

Fig. 4 is a view showing an image visually 
recognized by the observer. 

Figs. 5A, 58, 5C and 5D are views for illustrat- 
ing image processing. 

Figs. 6A and 6B are views for illustrating image 
processing. 

Fig. 7 is a view for illustrating the pixel arrange- 
ment of a combined image. 

Fig. 8 is a view for illustrating the pixel arrange- 
ment of the combined image. 

Figs. 9A and 9B are views for illustrating the 
pixel arrangement of the combined image. 

Fig. 10 is a view for illustrating a smoothing 
method. 

Fig. 11 is a view for illustrating the pixel ar- 
rangement of the combined image. 

Fig. 12 shows a second embodiment of the 
image display apparatus of the present invention. 

Figs. 13A, 13B and 13C are views for illustrat- 
ing the pixel arrangement of a combined image. 

Fig. 14 shows a third embodiment of the image 
display apparatus of the present invention. 

Fig. 15 shows a fourth embodiment of the 
image display apparatus of the present invention. 

Fig. 16 shows a fifth embodiment of the image 
display apparatus of the present invention. 

Fig. 17 is a view for illustrating the relations 
between a head portion position detection signal 
and a visual axis detection signal. 

Figs. 18A, 18B and 180 are views for illustrat- 
ing the relations between a detection signal and an 
image display state. 

Fig. 19A shows a first embodiment of the im- 
age photographing apparatus of the present inven- 
tion. 

Fig. 19B shows a modification of the first em- 
bodiment of the image photographing apparatus of 



the present invention. 

Fig. 190 shows a second embodiment of the 
image photographing apparatus of the present in- 
vention. 

5 

DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

Fig. 2 is a schematic diagram showing a first 

10 embodiment of the image display apparatus of the 
present invention. Light from a back light source 1 
such as a fluorescent lamp is modulated by an 
image display element 2 such as a liquid crystal 
display element, and for example, a beam of light 3 

75 emitted from a point a is displayed. This beam of 
light 3 passes through a half mirror 16, is con- 
verted into a beam of light 5 by a lens 4 and enters 
an observer's pupil 6. At this time, the power of the 
lens 4 and an optical device are set to suitable 

20 values, whereby the observer observes the beam 
of light 5 as a beam of light 7 emerging from a 
point a'. Likewise, a beam of light 8 from a point b 
on the image display element 2 is converted into a 
beam of light 9 by the lens 4, and the observer 

25 observes the beam of light 9 as a beam of light 10 
emerging from a point b'. 

Further, light from a back light source 12 is 
modulated by an image display element 13 such 
as a liquid crystal display element, and for exam- 

30 pie, a beam of light 14 emitted from a point c is 
displayed. This beam of light 14 is reflected by a 
half mirror 16 having drive means 15 movable on 
two axes, and enters the lens 4 and becomes a 
beam of light 17, which enters the observer's pupil 

35 6. Accordingly, the observer observes the beam of 
light 17 as a beam of light emerging from a point 
c'. Likewise, a point d on the image display ele- 
ment Is observed as a point d' on a virtual image 
plane 1 1 . 

40 As described above, the observer can observe 

while superposing the image information of the 
image display element as a virtual image 2' and 
the image information of the image display element 
13 as a virtual image 13* one upon the other on the 

45 imaging position 1 1 of the virtual image and enlarg- 
ing the two images. 

These two virtual images 2' and 13' form a 
combined image in which the different image in- 
formation 13* is superposed upon a cirtain portion 

50 on the image information 2' as the background, and 
the image information 13' is displayed at any loca- 
tion on the image information 2' by the half mirror 
16 being rotatively controlled on two axes by the 
drive means 15 controlled by a positional signal 

55 from a controller 26. 

The displayed position of the image informa- 
tion 13' will now be described. The image display 
apparatus according to the present invention has a 
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visual axis direction detecting unit 35. 

As the visual axis detecting method, use can 
be made of one of conventional methods such as 
the search coil method, the corneal reflection meth- 
od and the limbus tracking method, but the search 
coil method requires the mounting of a strong film 
contact lens (larger than ordinary contact lenses) 
having a coil wound therein and is unsuitable for 
practical use, and the limbus tracking method is a 
method of measuring by detecting the difference In 
the reflectance of Illumination light illuminating the 
vicinity of the boundary between the white (sclera) 
and the iris (cornea) of the eye between the re- 
spective portions, and has the advantage of being 
capable of constructing the system of an infrared 
light source and two photodetector elements, but 
suffers from a problem that measurement in the 
vertical direction is difficult. 

Therefore, the present embodiment adopts a 
detecting method using the corneal reflection meth- 
od which is simplest in construction and which has 
sufficient accuracy. 

In Fig. 2, the reference numeral 18 designates 
a light source such as a light emitting diode which 
emits invisible infrared light to the observer. The 
light source 18 Is disposed on the focal plane of a 
light projecting lens 19. 

The infrared light emitted from the light source 
18 is made into a parallel beam of light 20 by the 
light projecting lens 19, passes through a half mir- 
ror, Is reflected by the half mirror 16, and enters 
and illuminates the observer's pupil 6 through the 
lens 4. At this time, a corneal reflection image 
based on part of the infrared light reflected by the 
surface of the cornea (not shown) of the eyeball 
again returns along the aforedescribed optical path, 
is reflected by the half mirror 22 and becomes a 
beam of light 21, which is condensed by a light 
receiving lens 23 and forms a corneal reflection 
image on an image sensor 24. Also, beams of 
reflected light from the two end portions of the iris 
(not shown) likewise form an Image on the image 
sensor 24 by the light receiving lens 23. 

These unique points (the corneal reflection im- 
age and the end portions of the iris) are detected 
and subjected to a suitable calculation process, 
whereby the angle of rotation of the optical axis of 
the eyeball can be found. For example, by 

jS^L-slne = (Za + Zb)/2-Zd, 

the angle of rotation e of the optical axis of the 
eyeball can be found. In the above expression, /S is 
the Imaging magnification of each unique point 
onto the image sensor 24, L is the distance from 
the center of curvature of the cornea to the iris, and 
Za, Zb and Zd are the positions of the two end 
portions of the iris and the cornea reflection image, 



respectively, on the Image sensor 24. 

A visual axis detection signal from the visual 
axis direction detecting unit 35 constructed on the 
basis of the principle as described above is sub- 

5 jected to a suitable calculation process by a detec- 
tion circuit 25, and a visual axis direction output 
signal Is inputted to the controller 26. 

This visual axis direction signal is inputted as 
the positional signal of the virtual image 13' to a 

10 drive circuit 27, and the inclination of the half mirror 
16 is controlled by the drive means 15 rotatable in 
the directions of two axes (horizontal and vertical 
directions), and the virtual image 13' Is displayed 
always In coincidence with the visual axis direction. 

15 If the thickness of an optically transparent member 
constituting the half mirror 16 Is great, the dis- 
played position of the image 2' will be slightly 
moved by the rotation of this half mirror 16 and 
therefore, it is desirable that the thickness of the 

20 half mirror 16 be as small as possible. 

In the present embodiment, in order to prevent 
the beam of light from the back light source 12 or 
1 from entering the visual axis direction detecting 
unit 35 as an unnecessary beam of light, an optical 

25 filter absorbing or reflecting visible light and trans- 
mitting only infrared light therethrough can be pro- 
vided at a suitable location between the half mirror 
22 of the visual axis direction detecting unit 35 and 
the half mirror 16 to thereby prevent the incidence 

30 of unnecessary light onto the image sensor 24 and 
enhance the accuracy of the detection signal. 

Also, in the present embodiment, there has 
been shown only a case where only the visual axis 
direction detecting unit is provided to thereby con- 

35 trol the position at which the virtual image 13' is 
displayed, but it is also possible to control the 
displayed position of the virtual image 13' by the 
positional signal of the observer's head portion and 
the visual axis detection signal, by the use of a 

40 suitable head portion position detecting method 
(which will be described later In detail), for exam- 
ple, a method of forming a magnetic field around 
the head portion and providing a magnetic sensor 
on the head portion. 

45 Particularly, in the detection of the visual axis 

direction using the corneal reflection method, the 1 
mm movement of the reflected image corresponds 
to 10 degrees of eyeball motion and therefore, if 
the position of the head moves by 1 mm in parallel 

50 (without rotating), a detection error of 10 degrees 
will occur. Therefore, it is desired that the signal of 
the head portion position detecting means be used 
to cancel It. As a matter of course, use can also be 
made of a conventional method utilizing the re- 

55 fleeted Image from the rear surface of the cry- 
stalline lens (called the fourth Purkinje Image) to 
cancel the movement of the head. As will be de- 
scribed later, however, the present invention is 
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characterized in that the image information 2' as 
the background Is varied by the signal from the 
head portion position detecting nneans, and the 
head portion position is always detected from this 
character and therefore, a visual axis detecting 
method using a simple corneal reflection method 
using the first Purkinje image will suffice. 

As described above, the image display appara- 
tus according to the present invention has means 
for relatively moving the image information from 
the two image display elements 2 and 13 on the 
imaged position 11 of the virtual image, whereby 
different image information 13' can be displayed at 
any position on the image information 2' as the 
background. 

At this time, the image information 13' has its 
position controlled by the detection signal from the 
visual axis direction detecting unit 35 and Is dis- 
played always in coincidence with the visual axis 
direction. 

The image information 13' is part of the image 
information 2' and by making its image quality 
highly fine, the observer can highly finely observe 
the image information 13' in the portion of the 
image information 2' of a wide field angle the 
observer wants to see (the vicinity of the gaze 
point). 

Also, the image information 2' as the back- 
ground is varied by the signal from the head por- 
tion position detecting means and further, the gaze 
point on the image information 2' is extracted by 
the signal from the visual axis direction detecting 
unit and the display information of the image dis- 
play element 13 is changed, whereby the observer 
can move his or her head and eyes to thereby 
observe image information which is very wide, e.g. 
ranges over 360 degrees around the observer as if 
all of the image information was displayed highly 
finely. Of course, the order of the rewriting of the 
above-described two images is sometimes con- 
verse as when the gazing point has moved and 
thereafter the head turns in that direction. 

Generally, in equilibrium with yield, an LCD' 
having a small pixel size and a large display area 
is difficult and expensive to manufacture. In the 
present embodiment, the LCD 2 is an LCD having 
a large display area and a large pixel size, and the 
LCD 13 is an LCD having a small display area and 
a small pixel size, and when the display area of the 
LCD 2 is S2 and the display area of the LCD 13 is 
SI 3, the screen size of the LCD 2 and the screen 
size of the LCD 13 are approximately VS2 :VS13 . 
Also, the LCD 2 and the LCD 13 are optically at the 
same distance relative to the lens 4, and the ratio 
between the sizes of images 2' and 13' enlargedly 
projected onto the virtual image plane 11 are also 
approximately VS2 :VS13 . Accordingly, the ob- 
server can visually recognize the image of the LCD 



2 as a large image 2' of coarse pixels on the virtual 
image plane and the image of the LCD 13 as a 
small image 13' of fine pixels on the virtual image 
plane while superposing the two images one upon 

6 the other. 

Description will now be made of image signals 
sent to the LCD 2 and LCD 13 and an image 
visually recognized by the observer. 

Figs. 3A to 3D illustrate the images visually 

10 recognized by the observer in the present embodi- 
ment. Fig. 3A shows the original image information, 
and Fig. 3B shows image information of the original 
image information of Fig. 3A which is displayed by 
the LCD 13. When the cutting-out of the image is 

75 to be effected by the head portion position and 
direction detecting means, the image information of 
Fig. 3A becomes the image information of the 
widely displaying Image area 2*. 

The controller 26 shown in Fig. 2 receives the 

20 visual axis position signal from the detection circuit 
25 and cuts out of the original image information 
Fig. 3A, an image of a size corresponding to the 
display area of the LCD 13 when the visual axis 
position from within the original image information 

25 is the center and is imaged on the virtual image 
plane 11, and makes it into partial image informa- 
tion Fig. 3A. The controller 26 shown in Fig. 2 
sends the original image information Fig. 3A to a 
drive circuit 28 and sends the image information 

30 Fig. 3B to a drive circuit 29, and the drive circuits 
28 and 29 use the sampling or the like of image 
data by a method similar to that conventionally 
used for the driving of LCD to display images on 
the LCD 2 and LCD 13, respectively. 

35 Fig. 3C shows the image actually displayed on 

the LCD 2 shown in Fig. 2, and Fig. 3D shows the 
image actually displayed on the LCD 13, and the 
LCD 2 of Fig. 2 has large pixels and a large display 
area and therefore, Fig. 3C is a coarse image 

40 having a great angle of display field, and the LCD 
13 of Fig. 1 has fine pixels and a small display 
area and therefore. Fig. 3D, is a highly fine image 
having a small angle of display field. The drive 
circuit 27 of Fig. 2 effects the two-axis rotation of 

45 the half mirror 1 6 in accordance with the aforemen- 
tioned visual axis position signal, and gives the half 
mirror driving system 15 such a drive signal that 
the center of the virtual image of the LCD 13 by 
the lens 4 coincides with the observer's visual axis. 

50 Thus, the observer visually recognizes on the vir- 
tual image plane 11 the image of Fig. 3C displayed 
on the LCD 2 and the image of Fig. 3D displayed 
on the LCD 13 while superposing these two images 
one upon the other, and the center of the image 

65 Fig. 3D displayed on the LCD 13 coincides with the 
observer's visual axis. Fig. 4 shows the image 
visually recognized by the observer in the present 
embodiment, and the center of the virtual image of 
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the image displayed on the LCD 13 by the lens 4 
is coincident with the observer's visual axis. More- 
over, the images displayed on the LCD 2 and LCD 
13 are superposed one upon the other by the 
above-described processing and are visually recog- 
nized by the observer, and a highly fine image can 
be observed near the gazing point. 

If at this time, the drive circuit 29 and the 
illumination light source 12 are set so that the 
image to be displayed on the LCD 13 may be 
displayed with high luminance, the visual recogniz- 
ability of the highly fine image near the gazing 
point could be improved. Further, if such a mea- 
sure that the displayed image on the LCD 13 
gradually becomes lower in luminance from the 
center toward the marginal portion thereof is adopt- 
ed, the luminance difference in the boundary por- 
tion between the narrow image 13* and wide image 
2' projected onto the virtual image plane II could be 
made inconspicuous. Specifically, the drive circuit 
29 can be set such that the image signal of the 
image displayed on the LCD 13 becomes lower in 
luminance from the center toward the marginal 
portion, or an optical filter such as an ND filter 
which becomes higher in light absorption rate from 
the central portion toward the marginal portion can 
be installed in the LCD 13. 

As a basic method of solving the difficulty for 
the image to be seen in the boundary of the 
superposed portion of the images, there is a meth- 
od which will herein after be described, besides the 
above-described method. Figs. 5A to 5D, 6A and 
6B are views for illustrating the processing of the 
images in an embodiment of the present invention 
wherein overlap is prevented. However, in Figs. 5C 
and 5D and Fig. 6B, for illustration, pixels are 
depicted more coarsely than in the actual embodi- 
ment. 

Fig. 5A shows the original image information 
which is about to be displayed. Fig. 6A shows the 
image information sent to the LCD 2, Fig. 5B 
shows the image information sent to the LCD 13, 
and Fig. 5C shows the combined image visually 
recognized by the observer. In Fig. 6A, a black- 
colored portion (E) is a portion in which the LCD 2 
displays perfectly black (does not display any im- 
age). The original image information Fig. 5A is sent 
to the controller 26 of Fig. 2 by a signal line, not 
shown. The controller 26 receives the visual axis 
position signal from the detection circuit 25, divides 
the original image information Fig. 5A into the 
center as the visual axis position in the original 
image information, the image Fig. 5B to be dis- 
played on the LCD 13, and the image information 
Fig. 6A of said image in which of the original image 
information Fig. 5A, the portion corresponding to 
the image information Fig. 5B is a black signal 
(luminance signal 0), and sends them to the drive 



circuits 29 and 28 as the image signals B and A of 
Fig. 2, respectively. 

The drive circuits 29 and 28 effect the sam- 
pling or the like of image data by a method con- 

5 ventionally used for the driving of LCDs, and send 
the image signals B and A to the LCD 13 and LCD 
2, respectively, to thereby make the LCD 13 and 
LCD 2 display the images. As previously de- 
scribed, Fig. 6B shows the image actually dis- 

10 played on the LCD 2, and since the LCD 2 has 
large pixels and a large display area, this image 
becomes a coarse image great in the angle of 
display field. Fig. 5D shows the image actually 
displayed on the LCD 13, and since the pixels of 

75 the LCD 13 are small and the display area thereof 
is small, this image becomes a highly fine image 
small in the angle of display field. 

The image signal A sent from the controller 26 
to the drive circuit 28 can also be sent as an image 

20 signal compressed by the controller 26. 

In the present embodiment, by the fact that as 
previously described, the LCD 2 and the LCD 13 
are disposed optically at the same position relative 
to the lens 4 of Fig. 2 and by the driving of the half 

25 mirror 16, the image visually recognized by the 
observer becomes the image Fig. 5C comprising 
Fig. 6B and Fig. 5D superposed one upon the 
other. However, the center of the observer's visual 
axis is the point O of Fig. 5B, and this is substan- 

30 tially coincident with the center of the image Fig. 
5D actually displayed on the LCD 13 of Fig. 2. 

The center of the visual axis detected in the 
embodiment of Fig. 2 is one detected indepen- 
dently of the size and arrangement of the pixels of 

35 the LCD 2. Fig. 7 is a view for illustrating the size 
and arrangement of the pixels of the LCD 2 and the 
relation of the displayed position of the image 
displayed on the LCD 13 to the observer. 

In Fig. 7, in contrast with the pixels of the LCD 

40 2 of Fig. 2 represented in a lattice-like form, the 
image portion displayed on the LCD 13 is indicated 
by hatching. 

In the aforedescribed embodiment, the center 
of the visual axis is determined with the size and 

45 arrangement of the pixels of the LCD 2 of Fig. 2 
not taken into account and therefore, the boundary 
when the image information Fig. 5B to be dis- 
played on the LCD 13 and the image information 
Fig. 5A displayed on the LCD 2 are superposed 

50 one upon the other does not always coincide with 
the actual boundary between the pixels of the LCD 
2, but generally overlap with some of the pixels of 
the LCD 2. Accordingly, when the sampling point of 
the drive circuit 28 rests on the black signal in the 

55 image information Fig. 6A, i.e., the portion in which 
the luminance signal is 0, the black portion dis- 
played on the LCD 2 (the portion which displays no 
image) becomes larger than the portion overlap- 



8 



13 



EP 0 618 471 A2 



14 



ping with tiie LCD 13, and an unnecessary blacl^ 
line (a non-display portion) appears in the display 
boundary portion between the LCD 2 and LCD 13 
of the image visually recognized by the observer. 
To prevent this, the vertical and horizontal lengths 
of the display portion of the LCD 13 can be set to 
integer times as great as the vertical and horizontal 
length of a pixel of the LCD 2 and the size of the 
display portion of the LCD 2 can be set to integer 
times as great as the vertical and horizontal dimen- 
sions of the LCD 13, and the visual axis position 
signal of the visual axis sensor can be converted 
into a discrete value based on the pixel arrange- 
ment of the LCD 2, in the controller 26. 

Fig. 8 shows a case where, for example, the 
vertical and horizontal lengths of the display portion 
of the LCD 13 are set to two times and three times, 
respectively, as great as a pixel of the LCD 2 and 
the vertical and horizontal dimensions of the dis- 
play portion of the LCD 2 are set to six times each 
as great as the vertical and horizontal dimensions 
of the display portion of the LCD 13. When the 
output of the visual axis sensor comes into one of 
areas partitioned at the size of the image of the 
LCD 13 on the virtual image plane 11, relative to 
the images of the pixel arrangement of the LCD 2 
(broken-line lattices) on the virtual image plane 11 
and area groups (solid-line lattices) partitioned at 
the size of the image of the LCD 13 on the virtual 
image plane 11 (for example, the point A), it is 
judged that the observer is looking at the central 
point B in that area, and as previously described, 
the original image information Fig. 5A can be de- 
composed into the image information Fig. 6A and 
the image information Fig. 5C. The digitizing of the 
above-described visual axis output is effected by 
the use of a CPU installed in the controller 26 of 
Fig. 2 and an ROM having the positional data of 
the above-mentioned area groups recorded therein. 

At this time, by properly setting the size of the 
pixels of the image sensor 24 of the visual axis 
detecting unit 35 and the imaging magnification of 
the unique point of the human eye onto the image 
sensor 24 in conformity with the size at which a 
pixel of the LCD 2 is visually recognized by the 
observer, the calculation process can be simplified. 
In the present invention, the angle of rotation 6 of 
the optical axis of the eyeball is found from 

i3-L. sine (Za + Zb)/2-Zd 

as previously described. 

jS and the size of the pixels of the image 
sensor 24 can be set so that the values of Za, Zb 
and Zd for the angle of rotation of the optical axis 
of the eyeball corresponding to the size at which a 
pixel is visually recognized by the observer may be 
integer times as great as the pixels of the image 



sensor 24. 

Numerous methods of visual axis detection 
have been proposed besides the method described 
in the present invention, but for any of those meth- 

6 ods, the calculation process can be simplified by a 
similar method. Also, where as described above, 
the image by the LCD 13 for the gaze area is 
moved with the pixel pitch of the image by the LCD 
2 for wide area display as the unit, the driving of 

10 the half mirror 22 may be discrete driving cor- 
responding to a pixel of the image by the LCD 2, 
and can be effected by the use, for example, of a 
stepping motor or the like. 

The above-mentioned ratios between the verti- 

75 cal and horizontal lengths of a pixel of the LCD 2 
and the vertical and horizontal lengths of the dis- 
play portion of the LCD 13 and the ratios between 
the vertical and horizontal lengths of the display 
portion of the LCD 2 and the vertical and horizontal 

20 lengths of the display portion of the LCD 13 are 
values set for the simplicity of description and can 
actually be set to values conforming to the purpose 
of use. Also, in the present embodiment, there is a 
case where in the boundary between the displayed 

25 image on the LCD 2 and the displayed image on 
the LCD 13, there is a discontinuous variation in 
the pixel size and it is conspicuous. In order to 
prevent this, the displayed image is made coarser 
from the center toward the marginal portion of the 

30 displayed image on the LCD 13, whereby the vari- 
ation in the pixel size in the boundary portion can 
be made inconspicuous. 

Figs. 9A and 9B illustrate the above-described 
processing. Fig. 9A shows an image to be dis- 

35 played on the LCD 13 of Fig. 2, and Fig. 98 shows 
an image subjected to the above-described pro- 
cessing and made coarser toward the marginal 
portion thereof and actually displayed on the LCD 
13. In Fig. 98, thin solid-line lattices indicate the 

40 pixels of the LCD 13, thick solid lines indicate the 
center lines for indicating the center of the LCD 13, 
and two thin slant solid lines (A) and (B) indicate 
the boundary of the image represented by the thick 
line in Fig. 9A. As shown in Fig. 9B, the display of 

45 the displayed image is made coarser from the 
center toward the marginal portion of the LCD 13, 
and in the boundary between the displayed images 
on the LCD 13 and the LCD 2 of Fig. 2, the 
coarseness of the displayed image on the LCD 13 

50 is made substantially equal to the coarseness de- 
termined by the pixel size of the LCD 2, whereby 
the feeling of physical disorder by a sudden vari- 
ation in the pixel size can be mitigated. To carry 
out this processing, the sampling in the drive cir- 

55 cuit 29 of Fig. 2 can be effected so as to be coarse 
in the marginal portion of the LCD 1 3 of Fig. 2 and 
fine in the central portion thereof. 
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For the smoothing of the pixels, as will 
hereinafter be described, use may be made of a 
method whereby the image signal displayed on the 
pixels of the gazing point portion is made into a 
signal taking the weighted mean of the image sig- 
nal roughly sampled correspondingly to the size of 
the pixels of the widely displaying LCD 2 and the 
original image signal and display is effected with 
the weight of the image signal which has been 
sampled more roughly toward the marginal portion 
of the gazing point portion Image being made 
great. 

Fig. 10 is a conceptual view for illustrating the 
smoothing method. 

The reference numeral 2' designates an image 
area widely displayed by the LCD 2, and this 
image area is divided into pixels (a-i, A-j) cor- 
respondingly to the pixels of the LCD 2 (i = 1...4, 
j = 1...5). 

The reference numeral 13' denotes the image 
area of the gazing point portion by the LCD 13, and 
this image area is divided into pixels (b-i, B-j) 
correspondingly to the pixels of the LCD 13 
(i = 1...7, j = 1...7). 

Image signals corresponding to the central nine 
pixels {(b-3, B-3), (b-3, B-4), (b-3, B-5,), (b-2, B-3), 
(b-3, B-4), (b-3. B-5). (b-4, B-3). (b-4, B-4), (b-4, B- 

5) } in the image area of the gaze portion are the 
original image signal as it is intactly flowed, and a 
signal having taken the mean at the weight of 1 to 
1 of the original image signal and the roughly 
sampled image signal is flowed to sixteen pixels 
(b-2, B-2), (b-2, B-3), (b-2, B-4), (b-2, B-5), (b-2, B- 

6) , (b-3, B-2), (b-3, B-6), (b-4, B-2), (b-5, B-2), (b-5, 
B-6), (b-6, B-2), (b-6, B-3), (b-6, B-4), (b-6, B-5), (b- 
6, B-6)} in the image area of the gaze portion 
which surround the center, and a signal having 
taken the mean at the weight of 1 to 2 of the 
original image signal and the roughly sampled im- 
age signal is flowed to the most marginal twenty- 
four pixels {(b-1, B-1), (b-1, B-2), (b-1, B-3), (b-1, 
B-4), (b-1, B-5), (b-1, B-6), (b-1, B-7), (b-2, B-1), (b- 
2, B-7), (b-3, B-1), (b-3, B-7), (b-3, B-1), (b-3, B-7), 
(b-4. B-1), (b-4. B-7). (b-5, B-1). (b-5, B-7), (b-6. B- 
1), (b-6, B-7), (b-7, B-1), (b-7, B-2), (b-7, B-3), (b-7. 
B-4), (b-7, B-5), (b-7, B-6), (b-7, B-7)} in the image 
area of the gazing point portion. 

The roughly sampled image signal is an image 
signal displayed on that pixel of the widely display- 
ing image area 2' on which rests a certain point, 
e.g. the left upper point in the pixels of the gazing 
point portion 13' (for example, (a-2, A-2) corre- 
sponds to (b-4, B-2)). Where the left upper point in 
the pixels of the gaze portion rests on the bound- 
ary between the pixels of the widely displaying 
area 2', the weighted mean with the image signal 
displayed on that pixel of the widely displaying 
image area 2' on which rests the right lower point 



in the pixels of the gazing point portion. 

Also, design may be made such that the pixel 
size of the LCD 13 is actually small in the centra! 
portion and large in the marginal portion and 

5 gradually becomes larger from the center toward 
the marginal portion. Fig. 11 shows the sizes and 
arrangement of the pixels of the above-described 
LCD 13. The size of the pixels in the most marginal 
portion is set to substantially the same as the size 

10 of the pixels of the LCD 2 of Fig. 2. 

In the first embodiment, as previously de- 
scribed, the image information capable of being 
displayed highly finely over the whole display area 
is divided into the image information of a small 

15 field angle portion (the image displayed on the 
LCD 13 of Fig. 2) displayed highly finely and the 
image information of a great field angle portion (the 
image displayed on the LCD 2 of Fig. 2) displayed 
coarsely, and in the coarsely displayed image por- 

20 tion, the high frequency component of the image 
signal has been discarded by the sampling circuit. 
In such a method, a high specification is required 
of a TV camera or a computer for making the 
original image or recording means for recording 

25 image data, or much time and cost are required for 
making image Information. 

An improvement over the first embodiment in 
the above-noted point is a second embodiment 
shown in Fig. 12. This embodiment is almost the 

30 same as the first embodiment in apparatus con- 
struction, and differs from the first embodiment 
only in the method of making the original image 
and the method of controlling the image. In the 
description of this embodiment, portions differing 

35 from those of the embodiment of Fig. 2 will de- 
scribed preponderantly and portions common to 
those of the embodiment of Fig. 2 will be briefly 
described or not described. 

Fig. 12 is a block diagram for illustrating a 

40 signal processing portion in the second embodi- 
ment of the present invention, and in Fig. 12, the 
same members as those in the embodiment of Fig. 
2 are given the same reference numerals. 

The reference numeral 50 designates an image 

45 information generation unit comprised of a com- 
puter such as a CPU or a graphic work station, or a 
plurality of computers effecting inter-process com- 
munication or the like and effecting synchronized 
calculation processes. The image information gen- 

50 oration unit 50 causes an outside recording appara- 
tus, not shown, or a recording apparatus in the 
image information generation unit 50 to generate a 
gazing point portion image having data necessary 
for image information generation in the image dis- 
ss play area, such as, for example, the three-dimen- 
sional data of the object of display in the display 
area, and receiving the visual axis signal from the 
detection circuit 25 and to be displayed highly 
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finely on the LCD 13 about the visual axis, and the 
background image to be displayed on the LCD 2 
other than said gazing point portion image, in a 
gazing point portion's image generation unit 50-2 
and a background image generation unit 50-1 in 
the image information generation unit 50, with fine- 
ness necessary for the image displayed on each 
LCD, on the basis of data necessary for said image 
information generation, sends these images to the 
drive circuits 29 and 28, respectively, and causes 
the images to be displayed on the LCD 13 and 
LCD 2, respectively. The gazing point portion's 
image generation unit 50-2 and the background 
image generation unit 50-1 generate the image 
information to be displayed with the fineness deter- 
mined by the pixel sizes of the LCD 13 and LCD 2 
and the sizes of the display portions thereof, by the 
conventionally used graphic calculation. The gazing 
point portion's image generation unit 50-2 and the 
background image generation unit 50-1 each may 
be a processor or a computer effecting said cal- 
culation, or two programs running in parallel by a 
computer. 

The controller 26 receives a visual axis signal 
from the detection circuit 25, and sends a driving 
signal to the drive circuit 27 for rotating the half 
mirror 1 6 on two axes. 

In the present embodiment, the division of the 
image information Into the gazing point portion and 
the background portion can be accomplished by a 
calculation and operation similar to those in the 
division in the first embodiment of the present 
invention being performed within the image in- 
formation generation unit 50. Further, of the image 
to be displayed on the LCD 2 corresponding to Fig. 
6B, the image information corresponding to the 
portion (E) of the black signal (luminance signal 0) 
superposed by the image on the LCD 13 need not 
be generated at all, but a black display signal can 
be put into that portion. Also, in the present em- 
bodiment, as regards said gazing point portion's 
image and said background image, besides the 
process of separating basically one image field into 
the gazing point portion and the background por- 
tion as in the signal processing in the first embodi- 
ment and the above-described present embodi- 
ment, the process of allotting the gazing point 
portion and the background portion to the fore- 
ground and the background, respectively, is possi- 
ble. 

Figs. 13A to 13C illustrate the process of allot- 
ting the gazing point portion and the background 
portion to the foreground and the background, re- 
spectively. Fig. 13C shows an image which be- 
comes the background, and this Image Is gen- 
erated by the background image generation unit 
50-1 of Fig. 12. Fig. 13A shows the whole image 
information in the display area, and a and b in this 



figure indicate persons which become the fore- 
ground. In Fig. 13A, c indicates an area to be 
displayed on the LCD 13 about the visual axis 
detected by the detection circuit 25. When the area 

6 c overlaps the person a or b who is to become the 
foreground, the image of that portion of the fore- 
ground a or b falling within the area c Is generated 
by the gazing point Portion image generation unit 
50-2 of Fig. 12. The other portion of the area c than 

10 the foreground a or b is filled up with a black 
display signal. Fig. 13B shows the Image informa- 
tion generated in the gazing point portion image 
generation unit 50-2 by the above described pro- 
cess and sent to the drive circuit 29, and the 

75 hatched portions indicate the black display signal. 
The observer can observe the background image 
Fig. 13C displayed on the LCD 2 and the gazing 
point portion image Fig. 13B displayed on the LCD 
13 while superposing these images one upon the 

20 Other. When the area C does not overlap the fore- 
ground A or B, the gazing point portion image 
generation unit 50-2 of Fig. 12 generates a whole 
surface black display signal (luminance signal 0). 
Accordingly, the observer visually recognizes only 

25 the background image. 

In the above-described process, the back- 
ground image Fig. 13C displayed on the LCD 2 
does not include the process of black-displaying 
the portion thereof superposed upon the image 

30 displayed on the LCD 13 and therefore, the ob- 
server observes the two images while completely 
superposing them one upon the other. In this case, 
it is preferable that the whole of the background 
Image be displayed darkly as compared with the 

35 gazing point portion image. 

The above-described process is carried out by 
the Image information generation unit 50 of Fig. 12. 
Of course, the method already described can be 
carried out for the processing of the background 

40 image in the superposed portion of the gazing 
point portion image and the background image and 
the processing to the boundary between the gazing 
point portion image and the background image. 
Fig. 14 Is a schematic view showing a third 

45 embodiment of the image display apparatus ac- 
cording to the present invention, and in Fig. 14, the 
same elements as the elements shown In the first 
embodiment are given the same reference numer- 
als. This embodiment differs from the first embodi- 

50 ment in that relay lenses 31 and 32 are used and 
that intermediate imaging is effected, and is the 
same as the first embodiment in the other points. 

In Fig. 14, the image information lights from the 
image display elements 2 and 13 are imaged on an 

55 Intermediate imaging plane 33 by the relay lenses 
31 and 32, respectively. At this time, the intermedi- 
ate imaging of image information 13 has its imaged 
position on the intermediate imaging plane con- 



11 



19 



EP 0 618 471 A2 



20 



trolled by a half mirror 16 having drive means 15 
controlled by a controller and a drive circuit, not 
shown, so as to become the image information of a 
portion in which there is the intermediate imaging 
of image information 2, and the two types of image 
information are combined on the intermediate im- 
aging plane 33. By observing the thus combined 
image through the lens 4, the observer can ob- 
serve the two images on the virtual image plane 1 1 
while superposing them one upon the other. 

Again in the present embodiment, the visual 
axis direction detecting unit 35 is disposed at a 
suitable position, whereby as in the first embodi- 
ment, the displayed position of the image informa- 
tion on the image display element 13 is controlled. 
In this manner, the observer can observe highly 
fine image information displayed near the gazing 
point on the image information from the image 
display element 2 as the background. 

Also, what combines the two images is the 
Intermediate imaging plane 33, and this also leads 
to the effect that the amount of movement (the 
angle of rotation) of the movable half mirror 16 may 
be small. 

Also, in the present embodiment, the size of 
the display portion of the LCD 13, the focal length 
of the relay lens 32, the size of the pixels of the 
LCD 2, the focal length of the relay lens 31 and the 
focal length of the lens 4 are set such that the size 
of the virtual image of the LCD 13 for highly fine 
display by the relay lens 32 and the lens 4 be- 
comes integer times as large as the size of the 
virtual Image of the pixels of the LCD 2 for wide 
area display by the relay lens 31 and the lens 4, 
and if the visual axis position signal of the visual 
axis direction detecting unit 35 is converted into a 
discrete value based on the pixel arrangement of 
the LCD 2 by the controller 26 and the displayed 
image on the LCD 13 is moved, the boundary 
between the displayed image by the LCD 13 and 
the displayed image by the LCD 2 can be made 
coincident with the boundary between the pixels of 
the image by the LCD 2. 

Further, in the present embodiment, a screen 
can be provided at the location of the intermediate 
imaging plane 33 and the images can be combined 
on this screen and can be observed through the 
lens 4. In this case, the light of the combined 
image diffused by the screen can be observed and 
therefore, it becomes possible to make the N.A. of 
the lens 4 small, and this also leads to the effect 
that the downsizing of the apparatus can be 
achieved. In Figs. 2 and 14, for simplification, the 
lens 4 is shown as a single lens, but it is desirable 
in imaging performance that the lens 4 be a ce- 
mented lens comprising two achromatized lenses. 

Fig. 15 is a schematic view showing a fourth 
embodiment of the image display apparatus ac- 



cording to the present invention, and in Fig. 15, the 
same elements as the elements shown in the third 
embodiment are given the same reference numer- 
als. 

5 In this embodiment, as compared with the third 

embodiment, by the use of a combiner element 36 
such as a concave half mirror, other image in- 
formation such as a natural landscape and the 
virtual images from the image display elements 2 

10 and 13 can be superposed one upon another and 
three types of image information can be observed. 
In Fig. 15, the image information lights from the 
image display elements 2 and 13 are imaged on 
the intermediate imaging plane 33 by the relay 

75 lenses 31 and 32, respectively. The two types of 
image information combined on this intermediate 
imaging plane 33 are made into predetermined 
divergent light by the combiner element 36 com- 
prising a concave half mirror and enters the ob- 

20 server's pupil, and can be observed as a virtual 
image on the virtual image plane 11. 

In the present embodiment, a concave half 
mirror is schematically shown as the combiner 
element 36, but use can be made of various optical 

25 elements such as a plane half mirror, a combina- 
tion of a plane half mirror and an eyepiece, or an 
aspherlcal half mirror or a reflection type hologram 
lens element. 

Further, in the present embodiment, the imag- 

30 ing plane of the concave half mirror is curved and 
therefore, a so-called "gate optical system" ar- 
rangement is adopted so that the intermediate im- 
aging plane of the relay lenses 31 and 32 for the 
image information from the image display elements 

35 2 and 13 and the imaging plane of the concave half 
mirror 36 may be made coincident with each other, 
but these are suitably changed and designed in 
conformity with the characteristics of the various 
combiner elements as mentioned above. 

40 As described above, in the image display ap- 

paratus of the present embodiment, the observer 
can observe the images on the image display 
elements 2 and 13 at a time while observing an 
image such as a natural landscape through the 

45 combiner element 36, and can observe the image 
information from the highly fine image display ele- 
ment 13 having its display position controlled by 
the signal from the visual axis direction detecting 
unit 35 always in the visual axis direction. The relay 

50 lenses 31 and 32 are not the requisite constituents 
of the present embodiment. 

In the first to fourth embodiment, for example, 
single liquid crystal display elements have been 
described as being used as the image display 

55 elements 2 and 13, but it is also possible to use in 
the present invention an image display element 
provided with liquid crystal elements having R, G 
and B signals inputted thereto and displaying re- 
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spective color images, and combining and display- 
ing the color images by a dichroic mirror or the 
like, and a plurality of such liquid crystal elements 
and a color combining system may preferably be 
used particularly as the image display element 13 
which is desired to display highly finely. Further, in 
lieu of the image display element 13 and the back 
light source 12, use can be made of an image 
display mechanism for driving a one-dimensional 
LED array by a video signal, rotating the half mirror 
16 in synchronism with the video signal, scanning 
the LED light and displaying a two-dimensional 
image by the afterimage effect. Even if in this case, 
one-dimensional LED arrays for R, G and B, re- 
spectively, are used, the drive circuit therefor is 
simple and moreover, the half mirror 16 is used as 
two parts as means for scanning the light from the 
LED arrays and means for deflecting the position of 
the two-dimensional image as the afterimage and 
the other constituents are a few, and this leads to 
the effect that the downsizing and light weight of 
the image display apparatus according to the 
present invention can be achieved. 

Fig. 16 is a schematic view showing a fifth 
embodiment of the image display apparatus ac- 
cording to the present invention. This embodiment 
is an image display apparatus designed such that a 
parallax image based on the parallax of two eyes is 
observed by the respective eyes, whereby a ste- 
reoscopic image can be observed. 

The light from the back light source 1 is modu- 
lated by the image display element 2 such as a 
liquid crystal display element, passes through half 
mirrors 16R and 16L, is caused to enter respective 
eyes 6R and 6L by lenses 4R and 4L disposed in 
front of the two eyes, and forms virtual images 2*R 
and 2'L on the virtual image plane 11. A visual axis 
direction detecting unit, not shown, detects the 
visual axis directions 41 and 42 of the two eyes, 
and drive means 15R and 15L for the half mirrors 
16R and 16L are controlled by a controller and a 
drive circuit, not shown, and image information 
from image display elements 13R and 13L irradi- 
ated in directions 13*R and 13T by back light 
sources 12R and 12L is imaged as virtual images. 
At this time, the image display element 2 displays 
image information as the background, while the 
image display elements 13R and 13L display two 
slightly different types of image information as the 
parallax image of the image 13' near the gazing 
point e. Thus, the observer becomes able to ste- 
reoscopically observe the image of image Informa- 
tion 2' which is near the gazing point e the ob- 
server wants to see. Also, in the present embodi- 
ment, head portion position detecting means, not 
shown, which has not been described in detail in 
the first to fourth embodiments is mounted, for 
example, on the top portion of the apparatus. This 



position detecting means can utilize a suitable 
method, and is one which uses magnetism and 
which is called a Polhemus sensor, which Is based 
on the principle that magnetic fields produced 

6 time-divisionally from three coils orthogonal to one 
another which are called sources are measured by 
three similar coils (sensors) and the positions and 
postures of the sensors are found from the 3 x 3 
bits of information, and as other methods, mention 

10 may be made of an optical method of finding the 
position from the Images of two cameras by the 
principle of trigonometrical survey, an ultrasonic 
method of measuring the time until a signal from 
an ultrasonic transmitter arrives at a receiver with 

75 respect to a combination of several transmitters or 
receivers, and calculating the position therefrom, 
and a method using a mechanical link mechanism 
disclosed by I. E. Sutherland in Proc. of the Full 
Joint Computer Conference , vol. 33, pp. 757-764 

20 (1968), but in the present embodiment, use is 
made of a method of mounting on the head portion 
a magnetic sensor for forming a magnetic field 
around it and detecting the magnetic field. A suit- 
able calculation process Is carried out by the use 

25 of a head portion position signal from such head 
portion position detecting means and said visual 
axis direction detection signal and the result Is 
inputted to a controller, not shown, thereby varying 
the image on the Image display element 2 which is 

30 displaying the image information 2' as the back- 
ground. That is, when the observer is observing by 
moving his or her visual axis with his or her head 
portion remaining fixed, the image information 2' is 
not changed, but by moving or rotating his or her 

35 head portion, the observer changes the image in- 
formation 2' in conformity with the movement or 
rotation of the head portion. Therewith, the gazing 
point e is found by the visual axis direction detec- 
tion signal and the image information 13* near it is 

40 changed. 

Reference is now had to Figs. 17, 18A, 18B 
and 18C to describe the calculation process for the 
above-described head portion position signal and 
said visual axis direction detection signal and the 

45 relation with the then displayed images. 

Generally, when the gazing point moves to an 
object to be observed (an Image one wants to see), 
the eyeballs move correspondingly thereto, and 
instead of this, it is also possible to move the head 

50 portion to thereby follow the object. Usually, how- 
ever, one uses these two movements to make 
one's visual axis follow. 

Here, for simplicity, consider the matter with 
the rotation of center of the eyeball and the origin 

55 of the head portion position made coincident with 
each other. Strictly, the visual axis direction is also 
changed by the parallel movement of the head 
portion and therefore, it is necessary correct this 
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and moreover, the origin of the head portion posi- 
tion (sensor position) and the center of rotation of 
the eyeball never coincide with each other and 
therefore, it is necessary to calibrate the relative 
position thereof in advance. 

In Fig. 17, suppose an orthogonal coordinates 
system having, x as the front of the observer. Here, 
0 represents the rightward and leftward rotation of 
the head portion, <^ represents the vertical rotation 
of the neck, \f/ represents the vertical inclination of 
the neck, a represents the horizontal rotation of the 
visual axis, and ^ represents the vertical rotation of 
the visual axis. 

Also, in Fig. 17, the reference numeral 101 
corresponds to the virtual Image plane of Fig. 16 
and here, It schematically indicates a hemisphere 
always spaced apart by a predetermined distance 
R from the observer, but actually, the virtual image 
plane is a plane to which the direction 102 of the 
observer's head portion is a normal. As a matter of 
course, this plane (virtual image plane) is some- 
times not a strict plane depending on the aberra- 
tion characteristics of the lenses 4R and 4L of the 
display apparatus, or is sometimes consciously 
curved relative to the observer in order to give the 
so-called "surrounding effect". 

At this time, as shown, the observer is gazing 
at a point e at the left lower corner (leftward at an 
angle of a1 and downward at an angle of /31) with 
his head rotated leftwardly by 6^ from the front, his 
neck rotated upwardly by </>1 and moreover his 
neck inclined rightwardly by an angle of From 
the aforedescribed assumption, the horizontal 
movement of the head portion Is 0 and therefore, 
by converting a variation In the eyeball motion 
coordinates system by inclining the neck into the 
original orthogonal coordinates system, the direc- 
tion of the visual axis can be calculated and at the 
same time, the position and direction of the head 
can also be determined. 

Now, in the following of the visual axis, when 
the gazing point Is in an effective field of view 
(which is generally said to be 15 degrees to right 
and left), an image can be recognized by only the 
motion of the eyeball, and when the gazing point 
moves beyond this range, the movement of the 
head also becomes necessary. 

Accordingly, in the present embodiment the 
field angle of a highly fine image 13' near the 
gazing point Is 30 degrees and the field angle of an 
image 2' of a wide field angle is 80 degrees, but In 
an extreme case as shown In Fig. 18A (a case 
where the gazing point e is at the left lower corner 
of the image information 2* of a wide field angle), 
the angles a and /3 of the visual axis direction each 
are 25 degrees (here, for simplicity, the shapes of 
the images 2' and 13' are supposed as a square) 
and exceed the range of the aforementioned effec- 



tive field of view. Accordingly, to recognize the 
substance of the image 2' in detail, it is necessary 
to rotate the head portion in that direction so that 
the gazing point e may fall within the effective field 

5 of view. 

As a result, as shown In Fig. 18C, the image 
13' must become the image of the substantially 
central portion of the image 2' and therefore, In 
conformity therewith, the display image information 

10 of the Image 2' displaying the background image is 
changed. Fig. 18B showing the relation between 
the Images 2' and 13' schematically shows the 
image display state in the course wherein the ob- 
server rotates his head portion to change from the 

75 display state of Fig. 18A to the display state of Fig. 
18C. 

As described above, the image display appara- 
tus of the present embodiment can also detect the 
position of the head portion to thereby vary the 

20 image information 2' in conformity with a change in 
the position of the head portion and highly finely 
display the portion the observer wants to see (the 
vicinity of the gazing point). Of course, this Is also 
possible In the first to fourth embodiments. The 

25 present Invention Is characterized In highly finely 
displaying the portion of the image information 2' 
the observer wants to see (the vicinity of the gaz- 
ing point), and in the present embodiment, it is 
further possible to observe this portion the ob- 

30 server wants to see as a stereoscopic image. 

In such case, the image information to be dis- 
played on the image display elements 13R and 
13L can be made into parallax images correspond- 
ing to the respective eyes. 

35 Moreover, it is known that a stereoscopic Im- 

age is better in image quality than a plane image of 
the same frequency band. (See, for example, NHK 
Technical Institute, R&D , No. 15, pp. 69-75, 
September 1991.) Consequently, by constructing 

40 the Image display apparatus according to the 
present Invention, there is also provided the effect 
that the quality of the image In the vicinity of the 
gazing point can be further improved. Also, as a 
matter of course, if the apparatus construction de- 

45 scribed in the first to fourth embodiments is pre- 
pared in two sets and respective parallax images 
are displayed to the respective eyes, there could 
be an effect similar to that of the present embodi- 
ment. In such case, the image information 2' as the 

50 background can also be displayed as parallax im- 
ages to the respective eyes and at this time, there 
can be realized an image display apparatus by 
which the whole image field is telescopically ob- 
served and the vicinity of the gazing point can be 

55 observed as a more highly fine stereoscopic im- 
age. In each embodiment, for the area in which of 
the image information displaying a wide range as 
the background, the image information displaying a 

14 
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narrow range is superposed, there may be brought 
about a state In which nothing is displayed, or for 
the area in which both overlap, the same images 
may be displayed while being superposed one 
upon the other. 

Also, if two liquid crystal display elements of 
the same screen size and the same pixel density 
are used and the magnifications of the relay lenses 
corresponding thereto are changed, it will be possi- 
ble to create an image of relatively rough pixels 
showing the background and an image of relatively 
fine pixels corresponding to the vicinity of the gaz- 
ing point. In this case, the two liquid crystal display 
elements used are the same, and this is excellent 
in massproductivity. 

Description will now be made of a photog- 
raphing apparatus for the images to be displayed 
by the above-described image display apparatus. 

Fig. 19A Is a schematic diagram showing a first 
embodiment of the Image photographing apparatus 
according to the present invention. 

A TV camera 103 having a wide angle lens 104 
mounted thereon photographs a forward field of 
view 106 over a relatively wide range through a half 
mirror 105. Also, a TV camera 107 uses a lens 
such as a standard lens or a telephoto lens to 
photograph a forward field of view 109 of a rela- 
tively narrow field angle through a half mirror 105, 
and records an image in an image memory, not 
shown. In the present embodiment, the former field 
angle is set to about 80 degrees and the latter field 
angle Is set to about 30 degrees. At this time, a 
controller 121 for photo-taking direction has Input- 
ted thereto by a suitable method a photo-taking 
direction intended by a photographer, controls a 
goniostage 113 having a driving unit such as a 
pulse motor and an automatically rotatable stage 
114 by a controller 117 with the aid of respective 
drive circuits 116 and 115, and photographs an 
image 106 of a wide angle. 

An image at a position particularly observed by 
the photographer is controlled by a controller 120, 
and a goniostage 110 having a driving unit such as 
a pulse motor and an automatically rotatable stage 
111 are driven by drive circuits 119 and 118, 
respectively, to thereby Introduce a highly fine Im- 
age 119. 

When the photographing apparatus of the 
present embodiment and the display apparatus 
shown, for example, in Fig. 2 are used, there can 
be constructed a system very effective as a visual 
angle sensor in the teleexistence technique. That 
is, in the foregoing; the image photographed by the 
photographer's intention and moreover, only the 
portion observed by the photographer are recorded 
highly finely, but in the display apparatus of Fig. 2, 
the position of the head and the visual axis of the 
observer (the wearer of the display apparatus) are 



detected. This signal is inputted to the controller 
121 for photo-taking direction to thereby photo- 
graph an image of a wide field angle and a highly 
fine image, and the image of a wide angle (the 

5 output from the TV camera 103) is displayed on 
the image display element 2 of Fig. 2 and the 
highly fine image (the output from the TV camera 
107) is displayed on the image display element 13, 
whereby the observer can observe the image ar- 

10 ound the location at which the photographing ap- 
paratus of the present embodiment is disposed as 
if he was there. 

The drive means used in the photographing 
apparatus of the present invention are not limited to 

75 the goniostage having a driving unit such as a 
pulse motor and the automatically rotatable stage, 
but can also be realized by the use of the half 
mirror 105 having, for example, drive means 122 
capable of rotatlvely driven on two axes, and an 

20 example of such construction is shown in Fig. 19B. 
Of course, as the construction of such drive means, 
various constructions could be employed if means 
for photographing the Image of a wide angle and 
means for photographing the highly fine image can 

25 pan and tilt vertically and horizontally and further 
among them, the highly fine photographing means 
can pan and tilt vertically and horizontally. 

Fig. 19C Is a schematic diagram showing a 
second embodiment of the image photographing 

30 apparatus according to the present invention. 

Again in this embodiment, It is possible to 
Introduce an image reflecting the photographer's 
Intention by the use of the controller 121 for 
phototaking direction, but here, description will be 

35 made of a case where the image photographing 
apparatus of the present embodiment Is used with 
the image display apparatus shown in Fig. 4. 

The signals from the head portion position de- 
tecting means and visual axis detecting means, not 

40 shown, which have been described in connection 
with Fig. 16 are inputted to the controller 121 for 
phototaking direction on almost real time. In this 
controller 121, the movement of the head relative 
to the detection signal for visual axis direction Is 

45 corrected by the use of the head portion position 
detection signal to thereby find the direction in 
which the observer is facing and the direction in 
which the observer is gazing, and the whole 
photographing apparatus 112 Is turned to the dlrec- 

50 tion in which the observer is facing by the control- 
ler 117 and drive circuits 115 and 116, whereby an 
image 106 of a wide angle is photographed by the 
TV camera 103 having the wide angle lens 104 
mounted thereon. Also, in order to photograph Im- 

56 ages 109R and 109L as seen from the directions of 
the right eye and left eye by TV cameras 107R and 
107L, respectively, the directions of mirrors 126R 
and 126L rotatable on two axes are controlled by 

15 
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controllers 124R and 124L, respectively, on the 
basis of the signal from the visual axis detecting 
means. 

The images thus photographed are outputted 
from respective TV camera controllers 127, 128R 
and 128L and inputted to the aforedescrlbed dis- 
play apparatus. 

That is, the image 106 of a wide angle photo- 
graphed by the present apparatus is displayed on 
the image display element 2 of Fig. 16 and forms 
the virtual image 2'. Also, the images from the TV 
cameras 107R and 107L are displayed on the 
image display elements 13R and 13L, respectively, 
of Fig. 16 and become the virtual image 13'. At this 
time, as is apparent from Fig. 19C, parallaxes are 
created in the respective images. Accordingly, the 
observer can observe the image in the direction 
109 photographed by the photographing apparatus 
of the present embodiment as the stereoscopic 
image 13' by the use of the display apparatus of 
Fig. 16. At this time, as previously described, the 
image 2' may be low in pixel density, and if the 
image pickup element density of the TV camera 
1 03 is the same as that of the TV camera 1 07, the 
image transmission from the TV camera controller 
127 to the Image display apparatus can more en- 
hance the image compression rate than the image 
transmission from the TV camera controller 128 to 
thereby heighten the transfer rate. 

The image compressing method used here 
may be one of various methods including the con- 
ventional methods H.261, JPEG and MPEG. 

Further, to record the image in the present 
embodiment, the information of the direction in 
which the observer is facing when he sees by 
means of the display apparatus of Fig. 16 and the 
direction in which he is gazing is required and 
therefore, the signal from the head portion position 
detecting means and the visual axis detection sig- 
nal are recorded in advance on the respective 
images, and the images coinciding with the head 
portion position and the visual axis direction when 
the observer has observed wearing the display 
apparatus are random-accessed from an image 
memory 129, whereafter they can be inputted to 
the respective display elements of the display ap- 
paratus. 

This invention relates to an image display ap- 
paratus having first image generating means for 
displaying first image information by a plurality of 
pixels emitting light, second image generating 
means for displaying second image information by 
a plurality of pixels emitting light, the second image 
information being part of the first image informa- 
tion, visual axis detecting means for detecting an 
observer's visual axis direction, optical means for 
combining a first image from the first image gen- 
erating means and a second image from the sec- 



ond image generating means, moving the second 
image relative to the first image in conformity with 
an output signal from the visual axis detecting 
means and turning it to the observer's pupil, the 

5 marginal portion of the second image being made 
substantially coincident with the boundary between 
the pixels of the first image when the images are 
combined, and image display changing means for 
changing the first image information and the sec- 

10 ond image information on the basis of the output 
signal of the visual axis detecting means. 

Claims 

75 1. An image display apparatus comprising: 

first image generating means for display- 
ing first image information by a plurality of 
pixels emitting light; 

second image generating means for dis- 
20 playing second image information by a plural- 

ity of pixels emitting light, said second image 
information being part of said first image in- 
formation; 

visual axis detecting means for detecting 

25 an observer's visual axis direction; 

optical means for combining a first image 
from said first image generating means and a 
second image from said second image gen- 
erating means, moving said second image rel- 

30 ative to said first image in conformity with an 

output signal from said visual axis detecting 
means and turning it to the observer's pupil, 
the marginal portion of said second image 
being made substantially coincident with the 

35 boundary between the pixels of said first im- 

age when said images are combined; and 

image display changing means for chang- 
ing said first image information and said sec- 
ond image information on the basis of the 

40 output signal of said visual axis detecting 

means. 

2. An apparatus according to Claim 1, wherein 
the size of said second image is integer times 

45 as great as the interval between the pixels of 

said first image when said images are com- 
bined. 

3. An apparatus according to Claim 1, wherein 

50 said first image information in an area wherein 

said first image and said second image are 
superposed one upon the other when said 
images are combined is said second image 
information. 

55 

4. An apparatus according to Claim 3, wherein 
the area of said images superposed one upon 
the other is not displayed by said first image 
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generating nneans. 



7. An apparatus according to Clainn 1, wherein 
said first and second Innages are combined on 
an intermediate imaging plane through respec- 
tive relay lenses, are enlarged as virtual im- 
ages and are turned to the observer's pupil. 

8. An apparatus according to Claim 7, wherein 
said relay lenses have different imaging mag- 
nifications. 

9. An apparatus according to Claim 1, wherein 
said image display apparatus is provided cor- 
respondingly to each of the observer's left and 
right eyes. 

10. An apparatus according to Claim 9, wherein 
said second image generating means corre- 
sponding to the left and right eyes display 
parallax image information corresponding to 
the respective eyes. 

11. An apparatus according to Claim 9, wherein 
said first image generating means is used in 
common. 

12. An image display apparatus comprising: 

first image generating means for emitting 
light and displaying first image information; 

second image generating means for emit- 
ting light and displaying second image infor- 
mation, said second image information being 
part of said first image information; 

visual axis detecting means for detecting 
an observer's visual axis direction; 

head portion position detecting means for 
detecting the position of the observer's head 
portion; 

optical means for combining a first image 
from said first image generating means and a 
second image from said second image gen- 



erating means, moving said second image rel- 
ative to said first image in conformity with an 

output signal from said visual axis detecting 
means and turning it to the observer's pupil; 
and 

image information changing means for 
changing said first image information and said 
second image information on the basis of the 
output signal of said visual axis detecting 
means and the output signal of said head 
portion position detecting means. 

13. An apparatus according to Claim 12, further 
comprising an image memory having recorded 
therein the image all around a predetermined 
position, and wherein a part of said image 
memory is inputted as said first image in- 
formation to said first image generating means 
in conformity with the output signal of said 

20 head portion position detecting means. 

14. An apparatus according to Claim 12, wherein 
said first image information in an area wherein 
said first image and said second image are 

25 superposed one upon the other when said 

images are combined is said second image 
information. 

15. An apparatus according to Claim 14, wherein 
30 the area of said images superposed one upon 

the other is not displayed by said first image 
generating means. 

16. An apparatus according to Claim 12, wherein 
35 said first and second Images are combined on 

an intermediate imaging plane through respec- 
tive relay lenses, are enlarged as virtual im- 
ages and are turned to the observer's pupil. 

40 17. An apparatus according to Claim 16, wherein 
said relay lenses have different imaging mag- 
nifications. 

18. An apparatus according to Claim 12, wherein 
45 said image display apparatus is provided cor- 
respondingly to each of the observer's left and 
right eyes. 

19. An apparatus according to Claim 18, wherein 

50 said second image generating means corre- 

sponding to the left and right eyes display 
parallax image information corresponding to 
the respective eyes. 

55 20. An apparatus according to Claim 19, wherein 
said first image generating means is used in 
common. 



5. An apparatus according to Claim 1, wherein 
the pixel density in the central portion of said 
second image is higher than the pixel density 6 
of said first image and the pixel density in the 
marginal portion of said second image is sub- 
stantially the same as the pixel density of said 
first image when said images are combined. 

10 

8. An apparatus according to Claim 1, wherein 
the luminance in the central portion of said 
second image is higher than the luminance of 
said first image and the luminance in the mar- 
ginal portion of said second image is substan- 75 
tially the same as the luminance of said first 
image when said images are combined. 
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21. An image photographing apparatus for an im- 
age display apparatus for displaying, on the 
basis of output signals from visual axis detect- 
ing means for detecting the visual axis direc- 
tion of an observer who observes a first image 5 
from first image generating means for display- 
ing first image information and head portion 
position detecting means for detecting the po- 
sition of the observer's head portion, a com- 
bined image of said first image and a second io 
image from second image generating means 

for displaying second image information which 
is part of said first image information, compris- 
ing: 

first image pickup means for obtaining said 75 
first image information; 

second image pickup means for obtaining 
said second image information; 

means for controlling the photo-taking di- 
rection of said first image pickup means in 
conformity with the output signal from said 
head portion position detecting means; and 

means for controlling the photo-taking di- 
rection of said second image pickup means in 
conformity with the output signal from said 
visual axis detecting means. 

22. An apparatus according to Claim 21, further 
comprising means for transmitting the image 
information obtained from said first image pic- 
kup means to said image display apparatus 
while making the compression rate of said 
image information higher than that of the in- 
formation obtained from said second image 
pickup means. 

23. An apparatus according to Claim 21, wherein 
said image photographing apparatus is pro- 
vided correspondingly to each of the observ- 
er's left and right eyes. 

24. An apparatus according to Claim 23, wherein 
said first image pickup means is used in com- 
mon. 

25. An image display method comprising: 

the first image generating step of display- 
ing first image information by a plurality of 
pixels emitting light; 

the second image generating step of dis- 
playing second image information by a plural- 
ity of pixels emitting light, said second image 
information being part of said first image in- 
formation; 

the visual axis detecting step of detecting 66 
an observer's visual axis direction; 

the step of combining a first image from 
said first image generating step and a second 



50 



image from said second image generating 
step, moving said second image relative to 
said first image in conformity with an output 
signal from said visual axis detecting step and 
turning it to the observer's pupil, the marginal 
portion of said second image being made sub- 
stantially coincident with the boundary between 
the pixels of said first image when said images 
are combined; and 

the image display changing step of chang- 
ing said first images information and said sec- 
ond image information on the basis of the 
output signal of said visual axis detecting step. 

26. A method according to Claim 25, wherein the 

size of said second image is integer times as 
great as the interval between the pixels of said 
first image when said images are combined. 



27. A method according to Claim 25, wherein said 
first image information in an area wherein said 
first image and said second image are super- 
posed one upon the other when said images 
are combined is said second image informa- 

25 tion. 

28. A method according to Claim 27, wherein the 
area of said images superposed one upon the 
other is not displayed in said first image gen- 

30 erating step: 

29. A method according to Claim 25, wherein the 
pixel density in the central portion of said 

second image is higher than the pixel density 
35 of said first image and the pixel density in the 

marginal portion of said second image is sub- 
stantially the same as the pixel density of said 
first image when said images are combined. 

40 30. A method according to Claim 25, wherein the 
luminance in the central portion of said second 
image is higher than the luminance of said first 
image and the luminance in the marginal por- 
tion of said second image is substantially the 
45 same as the luminance of said first image 

when said Images are combined. 



31. A method according to Claim 25, wherein said 
first and second images are combined on an 
intermediate imaging plane through respective 
relay lenses, are enlarged as virtual images 
and are turned to the observer's pupil. 



32. A method according to Claim 31 , wherein said 
relay lenses have different imaging magnifica- 
tions. 
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33. A method according to Claim 25, wherein said 
steps are provided correspondingly to each of 
the observer's left and right eyes. 

34. A method according to Claim 33, wherein said 
second image generating steps corresponding 
to the left and right eyes display parallax im- 
age information corresponding to the respec- 
tive eyes. 

35. A method according to Claim 33, wherein said 
first image generating step is used in common. 



other is not displayed by said first image gen- 
erating step. 

40. A method according to Claim 36, wherein said 
5 first and second images are combined on an 

intermediate imaging plane through respective 
relay lenses, are enlarged as virtual images 
and are turned to the observer's pupil. 

10 41. A method according to Claim 40, wherein said 
relay lenses have different imaging magnifica- 
tions. 



36. An image display method comprising: 

the first image generating step of emitting 
light and displaying first image information; 

the second image generating step of emit- 
ting light and displaying second image infor- 
mation, said second image information being 
part of said first image information; 

the visual axis detecting step of detecting 
an observer's visual axis direction; 

the head portion position detecting step of 
detecting the position of the observer's head 
portion; 

the step of combining a first Image from 
said first Image generating step and a second 
image from said second image generating 
step, moving said second image relative to 
said first image in conformity with an output 
signal from said visual axis detecting step and 
turning it to the observer's pupil; and 

the image information changing step of 
changing said first image information and said 
second image information on the basis of the 
output signal of said visual axis detecting step 
and the output signal of said head portion 
position detecting step. 

37. A method according to Claim 36, further com- 
prising the image memory step of recording 
image information all around a predetermined 
position, and wherein part of the image in- 
formation recorded by said image memory 
step is inputted as said first Image Information 
to first image generating means in conformity 
with the output signal of said head portion 
position detecting step. 

38. A method according to Claim 36, wherein said 
first image information in an area wherein said 
first image and said second image are super- 
posed one upon the other when said images 
are combined is said second Image informa- 
tion. 

39. A method according to Claim 38, wherein the 
area of said Images superposed one upon the 



42. A method according to Claim 36, wherein each 
75 of said steps is provided correspondingly to 

each of the observer's left and right eyes. 

43. A method according to Claim 42, wherein said 
second Image generating steps corresponding 

20 to the left and right eyes display parallax im- 

age information corresponding to the respec- 
tive eyes. 

44. A method according to Claim 42, wherein said 
25 first image generating step is used in common. 

45. An image photographing method for an image 
display method of displaying, on the basis of 
output signals from the visual axis detecting 

30 step of detecting the visual axis direction of an 

observer who observes a first image from the 
first Image generating step of displaying first 
image information and the head portion posi- 
tion detecting step of detecting the position of 

35 the observer's head portion, a combined image 

of said first image and a second Image from 
second image generating means for displaying 
second image information which is part of said 
first image information, comprising: 

40 a first image pickup step for obtaining said 

first image information; 

a second image pickup step for obtaining 
said second image information; 

the step of controlling the photo-taking dl- 

45 rection of said first image pickup step in con- 

formity with the output signal from said head 
portion position detecting step; and 

the step of controlling the photo-taking di- 
rection of said second image pickup step in 

50 conformity with the output signal from said 

visual axis detecting step. 

46. A method according to Claim 45, further com- 
prising the step of transmitting the image in- 

55 formation obtained from said first image pickup 

step to an image display apparatus while mak- 
ing the compression rate of said Image in- 
formation higher than that of the Image in- 
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formation obtained from said second image 
pickup step. 

47. A method according to Claim 45, wherein each 
of said steps is provided correspondingly to 
each of the observer's left and right eyes. 

48. A method according to Claim 46, wherein said 
first image pickup step is used in common. 
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FIG. 3 A 
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FIG. 9 A 



FIG. 9B 
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